Papers with Language Identification
A reproduction of Apple’s bi-directional LSTM models for language identification in short strings (2021.eacl-srw)
Copied to clipboard
| Challenge: | Language Identification is the task of identifying a document’s language. |
| Approach: | They propose to use bi-LSTMs to identify language on very short strings such as text message fragments to perform automatic spell check. |
| Outcome: | The proposed model outperforms open-source language identifiers and its language identification mistakes are due to confusion between related languages. |
Collecting Code-Switched Data from Social Media (L18-1)
Copied to clipboard
| Challenge: | a new method to identify code-switched data from the web is needed . code-witching is defined as the tendency of bilinguals to switch between languages . |
| Approach: | They propose a method that automatically collects code-switched tweets from the web . they use crowd-sourcing to obtain language identifiers for a subset of 8,000 tweets . |
| Outcome: | The proposed method identifies tweets as code-switched in languages L1 and L2 . it is compared to a Spanish-English corpus of code-witched tweets . |
GLUECoS: An Evaluation Benchmark for Code-Switched NLP (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show multilingual contextual embedding models perform better on cross-lingual and multilingual tasks. |
| Approach: | They propose to evaluate multilingual contextual embedding models on multilingual data . they use language identification from text, POS tagging, Named Entity Recognition and Question Answering . |
| Outcome: | The proposed benchmark evaluates models on language identification from text, POS tagging, Named Entity Recognition, Question Answering and a new task for code-switching, Natural Language Inference. |
Language-Informed Beam Search Decoding for Multilingual Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Beam search decoding is the de-facto method for decoding auto-regressive Neural Machine Translation (NMT) models, but decoding multilingual NMT models produces off-target translations . |
| Approach: | They propose a general decoding algorithm incorporating an off-the-shelf Language Identification (LiD) model into beam search decoding to reduce off-target translations. |
| Outcome: | The proposed language-informed beam search improves +1.1 BLEU and +0.9 BLUE on WMT and OPUS datasets and reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% respectively. |